1. Field of the Invention
The present invention relates to a six-degree-of-freedom mechanical armature and an integrated software system with input and output capability for manipulating real and virtual objects in three-dimensional space, and for manipulating a scan plane in magnetic resonance imaging.
2. Description of Related Art
Advances in medical imaging technology, including computerized tomography (CT), magnetic resonance imaging (MRI), and positron emission tomography (PET), coupled with developments in computer-based image processing and modeling capabilities, have lead to significant improvements in the ability to visualize anatomical structures in human patients. Real-time MRI inherently has advantages over conventional MRI because of its capability for rapid visualization of any scan plane and interactive adjustment of location. Interactive MRI is particularly useful for selecting an oblique scan plane in coronary artery cardiac imaging (see, for example, Hardy et al., Magnetic Resonance in Medicine 40:105-111, 1998), Real-time MRI also provides visualization of the dynamic process of anatomical motion, such as arrhythmic cardiac motion and peristalsis in the abdomen, without requiring any type of respiratory or cardiac monitoring. Real-time MRI has also been used to guide and monitor interventional procedures (see, for example, Cline et al., Radiology 194: 731-737, 1995; Susie et al., Magnetic Resonance in Medicine 47:594-600, 2002).
Development of a 6-DOF system for the manipulation and representation of a scan plane is closely linked with recent developments in real-time MRI. During real-time MRI, the operator frequently needs to prescribe the scan plane by a sequential translation and/or rotation of the current scan plane. Typically, a Cartesian coordinate is attached to the image plane with the origin of the coordinate system at the center of the image, x pointing to the right, y upward, and z out of the image plane towards the user. The location and orientation of the scan plane are relatively independent. The center of the field-of-view can be changed by sliding in the x, y, and z directions of the image plane while maintaining orientation of the scan plane. Alternatively, the scan plane can be rotated along any x, y, or z axis without necessarily changing its location. The operation of scan plane prescription is therefore essentially a 6-DOF task, which generally is difficult to perform because of the high dimensionality of the required operations. Human observers normally cannot mentally decompose orientation into separate rotation axes (see, for example, Parsons, Journal of Experimental Psychology: Human Perception and Performance 21:1259-1277 (1995).
Typically, a flat-screen is the only resource available to graphically indicate the location and orientation of the scan plane, compounding the problem of scan plane prescription. A two-dimensional projection of a wire-frame representation of the scan plane is often not enough to indicate its location and orientation. Considerable mental processing is required for the operator to adequately visualize the results of a sequence of translations and/or rotations of the scan plane. Operators often acknowledge the loss of awareness of the reference frame during real-time MRI. For example, it is well known in the art that an operator may incorrectly report a visual impression that the scan plane should go deeper in order to better capture a structure, when in fact the scan plane should actually be shallower. Moreover, the operator can only be certain that the last executed prescription is correct when the most recent magnetic resonance (“MR”) image is displayed. This “try-and-see”, trial-and-error approach is time consuming and often causes frustration for human operators.
In order to overcome the limitations noted above, interest has developed in the design of more intuitive user interfaces. However, most of this work focuses on software development to provide graphical tools (see, for example, Debbins et al., Magnetic Resonance in Medicine 36:588-595, 1996; Kerr et al., Magnetic Resonance in Medicine 38:355-367, 1997). State-of-the-art scan plane prescription is relatively time consuming. Using a standard mouse for pointing and clicking, a typical prescription of a double-oblique imaging plane using a commercial ID-rive interface (General Electric Medical Systems, Milwaukee) requires about 20 seconds. During clinical procedures, the precise placement of several scan planes is made even more difficult because of other ongoing time-limited demands experienced by the operator. For example, during stress echocardiography the operator must potentially record a number of dynamic imaging events, including changes in myocardial wall motion and tissue blood flow, during a period of transient tissue ischemia.
To improve the efficiency of scan plane prescription, hardware devices have been adopted for MRI applications (see, for example, Hardy et al. Magnetic Resonance in Medicine 40:105-111 (1998). Although some currently used hardware devices such as the Spaceball are capable of providing 6-DOF input, their usage is non-intuitive, primarily because the direction and distance of 3-D translation is controlled by the force vector that the operator exerts upon the sphere. Similarly, the rotation is controlled by the torque. Furthermore, current hardware devices provide inadequate visual feedback about the spatial location and orientation of the current scan plane. Consequently, the operator does not have adequate spatial awareness and often is left with an unacceptable level of uncertainty concerning the next moving direction. Spaceball is an isometric device which is good for rate control, but not good for position control (see, for example, Zhai Computer Graphics 32:50-54 (1998). Spacemouse might have some potential for providing 6-DOF input, however, it suffers the same problem as a Spaceball, namely it returns back to a null rest station when user support is withdrawn.
The prior art does not disclose a method or device that is capable of providing input 6-DOF spatial information while also physically representing the spatial location and orientation of the object after the manipulation, which is also capable of automatically changing its position and gesture to imitate its virtual correspondence. The integration of these capabilities is essential for the manipulation of a virtual object in 3D space. The present invention discloses a system wherein an input device provides 6-DOF spatial information which is integrated with an output device for efficient and intuitive scan plane prescription.
Comparison with Existing Devices
The best known and simplest implementation of 6-DOF manipulation is provided by the graphical sliders available on commercial computer screens. In the conventional method of MRI scan plane prescription, each slider is dragged with the standard computer mouse with 2-DOF with three sliders devoted to offset the next scan plane along x, y and z axes relative to the current plane, and another three for the rotation angles along x, y, and z axes. However, these conventional methods have fundamental problems. First, an operator can manipulate only one degree at a time, which makes it impossible to execute a coordinated movement in 6-DOF space. Second, human operators generally cannot mentally decompose orientation into separate rotation axes (see, for example, Zhai, Computer Graphics 32:50-54, 1998). Consequently, given a target orientation, an operator cannot determine the rotation angles along each rotation axis required to reach the goal without first executing several “practice” attempts blindly. Third, since each scan plane is prescribed relative to the previous scan plane, the axes for rotation are not static and evolve with time. This time-dependence feature makes scan plane prescription even more difficult than the operations involved in a fixed coordinate system. The present invention overcomes all of these problems by enabling the operator to move in a coordinated manner all 6-DOF required for the prescription of a scan plane.
Several prior art methods prescribe a double oblique scan plane using a mouse as an input device based on a multi-step procedure. Typically, in the first step of the procedure, two points placed on a current image plane are connected as a line which determines an orthogonal plane to the current plane, and serves as the intermediate scan plane. After an image is obtained in the intermediate scan plane, the first step in the procedure is repeated to obtain the final scan plane, which may or may not be the correct final scan plane. In this prior art method for scan plane prescription, additional corrective steps may be required to achieve the correct final scan plane. Moreover, this method does not allow the user to manipulate in a coordinated manner all degrees-of-freedom at the same time. In addition, this method relies on the flat screen to display the location and orientation of the scan plane in 3-dimensional space. It is well known in the art that a flat screen is not sufficient in the depth-dimension and often induces visual ambiguity. The present invention overcomes the first problem and solves the second one by providing a physical representation of the scan plane relative to the patient coordinate.
A mouse is usually classified as a free-moving isotonic device wherein displacement of the device is typically mapped to a cursor displacement. An isometric device, by comparison, does not move but rather remains fixed relative to a desktop. In general, an isotonic device provides superior performance for positioning tasks compared to an isometric device (see, for example, Zhai, Computer Graphics 32:50-54, 1998).
Modifications to a standard mouse are known in the prior art which make it possible to input the third coordinate as well as incorporate 3-D rotations (see, for example, the “Bat” device disclosed by Ware, The Visual Computer, Vol. 6, pp 245-253, 1990). U.S. Pat. No. 5,503,040 to Wright discloses a computer interface device now commercially available as “Cricket”™ (Digital Image Design Inc New York http://www.didi.com/www/areas/products/cricket/ which includes a gimbal mounted handle having a plurality of input members for effectively communicating navigation and command signals to a computer. This invention provides an operator with 6-DOF for navigation within a virtual reality world while simultaneously enabling the operator to enter a series of commands in order to effectively communicate the operator's intentions to a computer to effect a change within a virtual reality world. Similarly, the MITS Glove™ designed by Zhai (Zhai, Human Performance in Six Degrees of Freedom Input Control, Ph.D. Thesis, University of Toronto, 1995) provides 6-DOF input control.
However, most of these modified high-dimensional “flying mice” are instrumented with a magnetic tracker for 6-DOF sensing, which makes them inaccurate in the environment of MRI. Another drawback is that the devices cannot remain at a particular location without support, which makes its difficult to resume an incomplete operation due to either fatigue or re-positioning of the hand.
U.S. Pat. Nos. 5,335,557, 5,729,249, and 5,805,137 issued to Yasutake disclose touch sensitive input control isometric devices that are now available commercially (“Spaceball”™, Spaceball Technologies.) These patented devices provide a family of controllers which incorporate multiple force/touch sensitive input elements to provide intuitive input in up to 6-DOF, including position and rotation, in Cartesian, cylindrical, or spherical coordinate systems. Six dimensions of input can be generated without requiring movement of the controller, which provides a controller suitable for controlling cursors and display objects in an interactive computer system. Positional information is obtained either by use of a “pushing” or “dragging” metaphor. Rotational information is provided by either a “pushing,” “twisting,” or “gesture” metaphor. The same sensor is used for both positional and rotational inputs, and the two are differentiated by the magnitude of the force applied to the sensor.
Spaceball™ devices have been used to prescribe the scan plane of MRI (see, for example, Hardy et al., Magnetic Resonance in Medicine 40:105-111, 1998). The scan plane is rotated on any axis by twisting the sphere around that axis and is translated in any direction by pushing the sphere in that direction. An alternative user interface strategy is provided by the multi-axis hand controller disclosed by U.S. Pat. No. 6,101,893 to Wergen, now marketed as “Spacemouse”™ by Logitech U.S.A. “Spacemouse”™ is an elastic device with a small range of movement (5 mm in translation and 4 degree in rotation). A multidimensional handle controlled without displacement is used for precisely positioned control and input. The actuating rod is selectively and simultaneously subjected to lateral pressure and to bending by a surrounding fist. The third dimension is controlled without displacement by the thumb, which acts on an additional sensor lever.
There are, however, significant limitations to the inventions embodied by “Spaceball”™ and “Spacemouse”™, including insufficient feedback to the user at the kinesthetic channel (see, for example, Zhai, Computer Graphics 32:50-54, 1998). For example, Spaceball™ is completely rigid, which presents a serious limitation because kinesthetic or proprioceptive feedback can be critical to the operator's control performance. A second limitation of Spaceball™ is that it returns to a null-position when released giving no feedback on the current location in 3-D space of the object under manipulation. The 6-DOF system disclosed by the present invention overcomes these problems by being more intuitive in manipulating the scan plane. In the method of the present invention, the armature device is capable of maintaining the current location and orientation of the scan plane to provide better spatial awareness for the operator. In addition, the armature device can be used according to the invention to automatically place the surface to reflect the prescribed virtual scan plane.
6-DOF Devices in the Prior Art
Exemplary of other multi-degree devices is the finger manipulable 6-DOF “Fingerball”™ input device disclosed in U.S. Pat. No. 5,923,318 to Zhai et al. “Fingerball”™ is a  6-DOF isotonic device that an operator holds and freely moves in real 3-D space to control the position and orientation of a virtual 3-D object. Zhai's invention provides an isotonic 6-DOF input device which includes a housing having a shape and dimension effective to permit an operator to grasp and manipulate the housing using the fingers of one hand. In one embodiment the housing encloses an interior cavity adapted to contain a position sensor. The entire housing is a pressure sensitive switch which is activated by the operator squeezing the housing with his fingers and/or thumb from any position on the outer surface of the housing. In a preferred embodiment the input control device is spherical in shape and has a textured outer surface adapted to prevent slippage in the operator's fingers. In addition to the large muscle groups of the shoulders, arm and hand, the input device makes extensive use of the small muscle groups of the fingers and thumb. However, unlike the present invention, the “Fingerball”™ device disclosed by Zhai et al. is not able to maintain its position when support is not provided.
U.S. Pat. No. 6,115,028 issued to Balakrishnan et al. discloses a device for the input of 3 spatial coordinates. Balakrishnan's invention provides a three dimensional input system using tilt, an input system for controlling the position or motion of a cursor, and three dimensions that use x, y, and z positions for inputting two coordinates and tilt in a plane (x-y or z-y) to input a third (and possibly a fourth coordinate). The input system disclosed in Balakrishnan et al. for controlling the position or motion of a cursor. The controlled cursor is moved about on a surface for inputting two of the dimensions and tilted to input the third. The amount or degree of tilt and the direction of tilt controls the input of the third dimension. The base of the hand held device is curved so that the device can be tilted even while it is moved in two dimensions along the surface of the tablet. Tilting can be along two orthogonal axes allowing the device to input four coordinates if desired. The coil can also have switched resistors controlled by mouse buttons connected to it which the tablet can sense being activated to allow clutching and selection operations like those of a conventional mouse. Although the “MicroScribe 3D digitizer”™ can simultaneously provide 6-DOF inputs, unlike the present invention it cannot statically maintain its position or orientation. Furthermore, unlike the mechanical armature device disclosed by the present invention, the “MicroScribe 3D digitizer”™ cannot be used as an output device to generate a physical representation of the position/orientation of a virtual object. Other examples of mechanical armature devices with 6-DOF include several force-feedback hand controllers that are capable of inputting spatial coordinate/orientation information and output force feedback. These devices are available commercially as “Freedom 6S Force Feedback Hand Controller”™ (MPB, Montreal, Canada) and “Phantom 6-DOF”™ (SenSable Technologies, USA).
U.S. Pat. No. 5,576,727 issued to Rosenberg et al. discloses an electromechanical human-computer interface with force feedback method and apparatus, which can provide commands to a computer through tracked manual gestures and also provide feedback to the operator through forces applied to the interface. The invention disclosed by Rosenberg et al. provides an operator manipulable object coupled to a mechanical linkage that is, in turn, supportable on a fixed surface. The mechanical linkage or the operator manipulable object is tracked by sensors for sensing the location and/or orientation of the object. A multi-processor system architecture provides a host computer system interfaced with a dedicated microprocessor that is responsive to the output of the sensors and provides the host computer with information derived from the sensors. The host computer has an application program which responds to the information provided via the microprocessor and which can provide force-feedback commands back to the microprocessor. The force feedback is felt by an operator via the user manipulable object. Although the invention disclosed by Rosenberg et al. provides 5- or 6-DOF force feedback control with the feature of static balance, it is distinguished from the present invention by the fact that it is incapable of automatically moving to a given position with a desirable orientation. In addition, not all of its joints can maintain balance.
U.S. Pat. No. 6,593,907 issued to Demers et al. discloses a tendon-driven serial distal mechanism for providing 3-DOF for a rotating handle. According to this invention, three stages provide a serial mechanical linkage between a handle and a platform, which may itself be moveable in three degrees of freedom. Each stage has an axis of rotation, and the three axes intersect. The first stage is mounted to the platform in such a way as to provide rotation about the first stage axis. The first stage carries the second, allowing the second stage to rotate about its axis. The second stage carries the third stage, allowing the third stage to rotate about its axis. The third stage is fixed to the handle, and the third stage axis passes along the length of the handle. Each stage has a sensor to measure its rotation, and a tendon means of transferring torque from a remote motor to torque about the rotation axis of the respective stage. The sensors have two limited angle ranges of measurement, about 110 degrees wide and on opposite sides of the rotation. The third stage has an auxiliary sensor, mounted in quadrature to the main third stage sensor and connected to an idler that carries the third stage tendon. The auxiliary third stage sensor measures angles of rotation that are not measured by the main third stage sensor. The two third stage sensors together provide continuous roll measurement about the third stage axis. However, unlike the present invention, the device invented by Demers et al. does not represent the position/orientation of the corresponding virtual object. Furthermore, unlike the present invention, the method disclosed by Demers et al. is not able to automatically position a real object in the real world.
U.S. Pat. No. 5,792,135 issued to Madhani et al. discloses an articulated surgical instrument for enhancing the performance of minimally invasive surgical procedures. The instrument has a high degree of dexterity, low friction, low inertia and good force reflection. A cable and pulley drive system operates to reduce friction and enhance force reflection, and a wrist mechanism operates to enhance surgical dexterity compared to standard laparoscopic instruments. The system is optimized to reduce the number of actuators required and thus produce a fully functional articulated surgical instrument of minimum size. The four actuators are coupled by the four cables to the wrist mechanism, the rotary joint and the linear joint such that selective actuation of the actuators operates to move the first work member of the surgical end effector about two orthogonal axes with two degrees-of-freedom relative to the support member, extend and retract the support member along the support axis relative to the support bracket and rotate the support member about the support axis relative to the support bracket and thereby move the first work member of the surgical end effector relative to the support bracket with four degrees-of-freedom.
U.S. Pat. No. 6,394,998 issued to Wallace et al. discloses surgical instruments for use in minimally invasive telesurgical applications. The instruments include a base whereby the instrument is removably mountable on a robotically controlled articulated arm. An elongate shaft extends from the base. A working end of the shaft is disposed at an end of the shaft remote from the base. A wrist member is pivotally mounted on the working end. At least one end effector element mounting formation is pivotally mounted on an opposed end of the wrist member. A plurality of elongate elements, e.g., cables, extend from the end effector element mounting formation and the wrist member to cause selective angular displacement of the wrist member and end effector mounting formation in response to selective pulling of the elongate elements.
U.S. Pat. No. 6,441,577 issued to Blumenkranz et al. discloses techniques and structures for aligning robotic elements with an internal surgical site and each other. Manually positionable linkages support surgical instruments. These linkages maintain a fixed configuration until a brake system is released. While the brake is held in a released mode, the linkage allows the operating room personnel to manually move the linkage into alignment with the surgical site. Joints of the linkage translate the surgical instrument in three dimensions, and orient the surgical instrument about three axes of rotation. Sensors coupled to the joints allow a processor to perform coordinate transformations that can align displayed movements of robotically actuated surgical end effectors with a surgeon's hand inputs at a control station.
Applications to MRI
Motion artifacts due to normal or abnormal respiratory movements can degrade image quality in MR scans. Motion artifact suppression techniques have been useful in coronary artery imaging and in monitoring of heart wall motion, which is useful to assess the severity and extent of damage in ischemic heart disease. MR imaging of the coronary arteries, or MR angiography (MRA), has typically been performed using a technique to limit the MRI acquisition to avoid motion artifacts. Such techniques include requiring the patient to withhold breathing during the imaging, using oblique single-sliced image techniques, or respiratory-gated 3-D imaging techniques. However, repeated breath holding may not be feasible for many coronary patients and navigation techniques to-date have not generally provided a robust method which works over a range of different breathing patterns in a variety of patients. Another drawback to these approaches is that success or failure is usually not apparent for some time after the start of imaging, and many times not until the imaging has been completed.
Another application of the scan plane and image navigation method disclosed by the present invention relates to myocardial perfusion imaging to detect the passage of a contrast agent through muscle tissue in the heart and to study blood flow in the micro-circulation of the heart non-invasively. Typically, perfusion imaging consists of using injected contrast agents together with rapid imaging during the first pass of the contrast agent through the microvasculature with carefully optimized pulse-sequence parameters. Quantification of blood flow from these images is carried out with a region of interest-based signal, time-intensity curve analysis. To avoid cardiac motion artifacts, the perfusion images are typically acquired with ECG gating. However, since the period of image acquisition is usually one to two minutes long, the images suffer from significant respiratory motion artifacts. This then requires a manual registration and analysis of the perfusion images, which is cumbersome and time-consuming because the user must carefully arrange each image to compensate for the respiratory motion before proceeding to a region of interest time-intensity analysis.
A key requirement in minimally invasive procedures is to integrate the positioning of instruments, needles, or probes with image guidance to confirm that the trajectory or location is as safe as possible, and to provide images that enhance the ability of the physician to distinguish between normal and abnormal tissues. In interventional MRI applications, instruments must be positioned accurately within the field of view (FOV) or near the FOV of image acquisition. Placement may require acquisition of static images for planning purposes, either in a prior MRI examination or during the interventional MRI session, or real-time images in arbitrary scan planes during the positioning process. (See, for example, Daniel et al. SMRM Abstr. 1997, p. 1928; Bornert et al. SMRM Abstr. 1997, p. 1925; Dumoulin et al., Mag. Reson. Med. 1993, 29: 411-415; Ackerman et al., SMRM Abstr. 1986, p. 1131; Coutts et al., Magnetic Resonance in Medicine 1998, 40:908-13. One useful application of the present invention is to manipulate a virtual or real 3-D object, such as, for example, an ultrasound transducer to a position and rotate it to a desirable orientation corresponding to an MR scan plane position. Examples of other interventional MRI procedures that would benefit from the present invention include image-guided interstitial probe placement to provide high-temperature thermal therapy, cryotherapy, or drug therapy for tumors; localization of non-invasive focused ultrasound probes below the tissue surface for thermal therapy; and subcutaneous or transdural placement of biopsy needles or surgical instruments for minimally-invasive surgery.
For interventional MRI applications, there is the additional need to register data from other imaging modalities to provide comprehensive and complementary anatomical and functional information about the tissue of interest. Registration is performed either to enable different images to be overlaid, or to ensure that images acquired in different spatial formats (e.g., MRI, conventional x-ray imaging, ultrasonic imaging) can be used to visualize anatomy or pathology in precisely the same spatial location. While some algorithms exist for performing such registrations, computational cost would be significantly reduced by developing technology that enables data from multiple different imaging modalities to be inherently registered by measuring the patient's orientation in each image with respect to a common coordinate system.